Manage TAR Projects
OPEN DISCOVERY Review’s Technology-Assisted Review (TAR) is based on IPRO Intelligence and IPRO Eclipse Analytics. These advanced technologies provide a computer-learning process regarding document responsiveness that can help your firm improve review accuracy and lower review costs and risks.
Particularly beneficial for cases that contain large document sets, TAR combines with human expertise to allow documents to be classified more efficiently and accurately than with a human-only review.
TAR project management is straightforward. After a project is created, Eclipse automatically creates all of the components needed for the TAR document evaluation, thus providing a streamlined review workflow process. TAR-specific components that are created include:
|
|
|
Workflow Process
A TAR project includes three review phases, as explained in the following table. The flowchart in the section “The TAR Project” shows the basic workflow for a TAR project.
TAR Project Phases
The Analytics Engine
OPEN DISCOVERY Review directly integrates with the Analytics engine. Through the Technology Assisted Review (TAR) project process, the Analytics system can take an entire collection of data and automatically group it into multiple buckets, quickly organizing the data in a logical fashion based on conceptually related terms, topics, and clusters.
The TAR Project
The following diagram shows the workflow of how OPEN DISCOVERY Review handles the TAR process to effectively train the system and reduce the amount of validation review rounds.
The TAR process begins by creating a sample set of documents that are used to train the system. Analytics accelerates the training process and reduces the number of validation rounds by organizing the data into clusters that are topically related. It creates the initial training set by statistically and strategically pulling documents from cluster centers, providing the expert reviewer with a conceptually diverse group of documents so that it gains the most insight from the first round of review decisions.
The training process is then conducted by human expert reviewer(s). The reviewer(s) can code or tag the documents for the system to use as an example to later retrieve similar documents.
Additional issue coding with sub-category tagging can be set up at the same time without impacting the TAR results. The issue tags can be created within the same tag window and applied to all documents.
Pre-coded documents can be submitted to the TAR project after the first system generated training round is created and completed. The first system generated training round initiates the TAR prediction process. The training round also assists in determining prevalence (that is, an estimated percentage of how much relevant data exists in a data population). Prevalence is recorded in the TAR Project Summary Report as Training Set Responsive Percentage.
Performing additional training rounds up front (providing the system with more responsive examples) will improve the training model.
Once training is completed, the system then categorizes the rest of the documents in the TAR population and initiates a validation round. For validation, TAR uses statistical sampling to create a new round of the documents that have yet to be reviewed. To fine-tune and continue to train the system, the case team reviews this set of documents. Without knowing which category the system has grouped the documents into, the case reviewers tag each document as Relevant or Not Relevant. After a validation round is completed, review of TAR reports is next.
Analysis of the TAR Project Summary Report and TAR Validation Details Report determines whether the system has achieved optimum results.
Recall is a metric representing how much relevant data the system is finding. IPRO Review uses a 0.7 Recall to determine a “cut-off” point. Documents above the Relevancy Score cut-off is predicted relevant, and any documents below the cut-off is predicted non-relevant.
Precision is a metric representing how well the system is identifying relevant documents. With consideration of the project goals, review of the recall, precision, and Current Estimated Responsive count in the reports can determine when optimal results are achieved.
An additional round of validation may be suggested if goals are not met. If so, expert reviewers then review another set of documents to further refine the system’s knowledge to accurately code the rest of the document population. This iterative process continues until enough rounds have been completed for the system to achieve optimal results.
As an optional step, TAR can also be used to forgo reviewing documents that are likely Not Relevant. It is recommended to use an additional step to Certify the review by performing an additional validation round on documents that have been categorized by the system as Not Relevant. This process measures the accuracy of the review. IPRO Review provides several automated reports for audit purposes and to keep track of the TAR project status. The reports can be used for identifying the discrepancy documents, viewing the overall status, auditing each round of validation performed, as well as the statistically expected outcome.
Insights
Although TAR can be used to prioritize the review, it is also used to only review the documents that are categorized as relevant, ignoring the rest of the collection. When that method is used, it must be determined when to terminate the review. This hinges on the proportionality considerations outlined in (U.S.) Federal Rules of Civil Procedure 26(b)(2)(C) and 26(g)(1)(B)(iii), which, respectively, limit discovery if “the burden or expense of the proposed discovery outweighs its likely benefit.”
Related pages: